“The extent to which we can infer that the independent variable caused the dependent variable.”
External validity
“The extent to which the findings will generalize to other populations, settings, measures, and treatments.”
Measurement validity
“The quality of accuracy of individual measures or scores. The extent to which a score measures what it was intended to measure.”
Also measurement reliability
Internal Validity
“The extent to which we can infer that the independent variable caused the dependent variable.”
Three criteria for causality
IV must precede the outcome variable
IV must be related to the outcome
There must be no other variables that could explain why the IV is related to the outcome
Establishing internal validity by the research approach
Hierachy
Randomized Experiments
Quasi-Experimental studies
Comparative
Associational
Descriptive
Internal Validity is equivalence and control
Evaluating the internal validity of a study –
Equivalence of the groups on participant characteristics
Control of extraneous experiences and environmental variables
Establishing equivalence
Are groups equivalent prior to introduction of IV?
Assured without further work in randomized studies
Empirical comparisons for non-randomized studies
Can you use matching or statistical adjustments?
Establishing control
Extraneous and environmental variables
Not of direct interest
Influence the outcome
Imbalanced
Example: contamination
Is one group affected more than the other?
Less of an issue for laboratory studies
Other threats to internal validity
Regression to the mean
Dropouts/attrition
Bias in assignment
Carryover effects
Changes in environment
Instrument or observer inconsistency
Patient expectations
Observer bias
Break #1
What you have learned
Internal validity
What’s coming next
External validity
External Validity
Population external validity
Representative sample
Few or no restrictions
Few or no dropouts
Ecological external validity
Naturalness
Setting
Procedures
Distinction between internal and external validity
Sampling process influences external validity
Random samples versus non-random samples
Treatment allocation influences internal validity
Randomized design versus non-randomized design
Trade-offs between internal and external validity
High degree of control
Avoid issues of equivalence
Unnatural setting
Low degree of control
More chances of contamination
Closer to how medicine is practiced.
Break #2
What you have learned
External validity
What’s coming next
Measurement validity
Measurement quotes (1 of 2)
“The government is extremely fond of amassing great quantities of statistics. These are raised to the Nth degree, the cube roots are extracted, and the results are arranged into elaborate and impressive displays. What must be kept ever in mind, however, is that in every case, the figures are first put down by a village watchman, and he puts down anything he damn well pleases.”